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Orientation of Program 

This institute was designed to provide thorough and practical 
training in research, automatic data processing, program evaluation 
and review technique. The institute also provided for a general 
orientation and overview of Educational Research Information Centers 
(ERIC), Regional Educational Laboratories, and Research and Develop- 
ment Centers. 

The participants for the institute consisted of state educational 
agency personnel. The institute was sponsored by the Division of State 
Agency Cooperation and the Division of Research Training and Dissemina- 
tion of the U.S, Office of Education. The trainee group accepted 
consisted of nineteen state educational agency personnel. The educational 
background and professional experience of the group was quite varied. 

Ten members of the group presently had research responsibilities for 
their respective state departments. The other nine were responsible for 
a variety of programs such as: adult education, teacher certification, 

finance, and safety education. 

The duration of the inst .tute was for four weeks, consisting of 
two 2-week sessions. The first two week session began on March 25, 1967, 
and ended on April 7, 1967. The second two week session began on May 1, 
1967, and ended on May 12, 1967 » 
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Description of the Program 

The schedule of instruction consisted of two daily sessions 
of three hours each* The class presentations were basically lectures 
by the Instructors with adequate time for discussions. Practical 
exercises and problems were presented periodically in the form of 
hypothetical situations for the trainees to do during classtime and as 
a homework laboratory experience c The problem centered approach of 
instruction was used throughout the institute* 

The first two week session included Measurement in Educational 
Research * Research Design * and Program Evaluation * The second two 
week session of the institute included five days on Program Evaluation 
Review Technique (PERT), three days on Automatic Data Processing * one 
day on Educational Research Information Centers (ERIC) , and one day 
on Regional Educational Laboratories and R and D Centers. 

The curricula was adapted to the specific needs of the partici- 
pants and to che objectives of the institute* The specific course 
content for the various areas follows: 

1. Me asurement in Educational P^esearch * The material covered 
Included the meaning, characteristics, and processes of educational 
measurement and evaluation; statistical concepts; development and 
interpretation of standardized instruments, statistical tools and 
processes for testing validity and reliability; and instrument construc- 
tion and interpretation* 
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2. Research Design . The Research design content included the 
basic principles of design and methodology; problems and hypothesis; 
variables and definitions; importance and uses of sampling and 
randomness; survey, descriptive, observational and other research 
methods and applications. 

Program Evaluation , The program evaluation content included 
methodology and interpretation. The methodology covered theory, des- 
criptive statistics, correlations, analysis of variance, and significance. 
The interpretation covered statistical presentations and analysis, 
interpretations of research data, and conclusions. 

On a topical basis the following items were included in the 
first two week session by the instructional staff: 

Role of research and education 
Basic statistical concepts 
Measure of variability 
Skewness 

Normal distribution 
Research resources 

Norms and measurement of achievement 
Grade equivalents 

Standard scores (Z-scores, T-scores) 

Problems of interpretation 
Correlation 



Regression 



Intelligence assessment 
Sampling 

Confidence intervals 

Statistical significance 

Type I and II errors 

Analysis of Variance 

Factual designs interaction 

Differences in proportion (chi square) 

Factor designs 
Factor interaction 

Analysis of covariance and quasi-experimentation 

Measurement of affective variables 

Self-report documents 

Opinionaire and questionnaire 

Semantic differential 

Q-sort 

Social Measurement-resources 
Anxiety measurement 
Creativity 

Pre-experimental and quasi-experimental design 
Sources of internal invalidity of experiments 
Sources of external invalidity 

Internal and external validity of selected experiments 
Causation and correlation 

True experiment design and quasi-experimental design 
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Esvpsriiaental unit vs. unit of analysis 

Elements, operation and activities of the evaluation of educational 
programs 

Types of evaluation activities: formative vs. summative; 

instrumental vs. consequential 

Analysis of the nature of evaluation plans in a sample of 484 
Title I proposals 

4. Program Evaluation Review Technique (PERT). Five days of the 
institute were devoted to Research Management with the major emphasis 
being on P.E.R.T. The Program Evaluation and Review Technique included: 
an orientation of the development of a systematic method of devising a 
program plan, checking the logic of it, and keeping track of it in 
operation. The instruction included an analysis and application of the 
network concept and flow chart techniques. The establishment of the cost 
base and programmed budgeting was also a part of the curriculum content. 

On a topical basis the following items were included in the five 

day session on PERT: 

Orientation to Research Management 

Nature of management 

Management process 

Management information systems 

PERT history 

Establishing information base 



Introduction to PERT 
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Network construction 

Establishing time estimations 

Network analysis 

Scheduling the project 

Resource allocation 

Establishing the cost base 

PERT/cost 

Program budgeting 

Computer processing of Base Data 

Controlling as a Management Function 

Up-dating 

Management reports 
Problem identification 
Decision making 

Application of management information systems to education 
PERT implementation 

5, Automatic Data Processing. The three days spent on automatic 
data, processing as it relates to educational research included: general 

characteristics, principles and concepts; programming, functional 
operations, statistical calculations, and data analysis* 

On a topical basis the following items were included in the 
sessions on automatic data processing: 

Introduction to data processing in education 
Administrative uses 



Teaching A«D*^. 
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Computer assisted instruction 
Introduction to electro-mechanical machines 
The punched card 
Key punch 
Verifier 
Sorter 

Accounting machine 
Interpreter 
Reproducer 
Collator 

Introduction to computers 
Computer programming 
Systems development 
Research applications 
Local school relations 

6. Educational Research Information Center (ERIC). The institute 
provided for a one day general orientation and overview of E.R.I.C, This 
overviev? included a general overview of the organization and function of 
E.R.I.C. and the specific operation of the E.R.I.C. Clearinghouse for Rural 
Education and Small Schools located at New Mexico State University, Las 
Cruces, New Mexico. 

On a topical basis the following items were included in the session 
on ERIC: 

Organization of ERIC 
Objectives 

Facilities and equipment 
Documents and materials 

Operational procedures of a clearinghouse 
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7. Regional Educational Laboratories , The institute program 
p^jQvided for a one^day general orientation and overview of the 
organization, function, and operation of the established Regional 
Educational Laboratories. Specific information concerning the program 
of activities of the various established laboratories was presented. 

On a topical basis the following items were included in the 
session on Regional Educational Laboratories: 

The establishment of regional laboratories 
Function sample organizational structures 
Financing 

Operational procedures 
Educational change 

Relationships and roles of the Regional Laboratories to: 

state departments of education ^ local school districts, 
higher education, and R. and D. Centers 
The program was carried on in accordance with the approved 
proposal. There were no major changes in organization, operational 
procedures, or curricula content. The instructional staff made minor 
modifications wherein the background of the participants indicated that 
modifications x^ere desirable. The organization and materials covered 
accomplished the objectives of the institute. 

Evaluation o f the Program 

The evaluation of the program x^as based on (1) pre-tests and 
post-tests, and periodic mastery tests of the understandings of 
materials covered, (2) review of assignments given in work sessions 
and take home exercises, (3) evaluative reports by the trainees. 
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(4) general observations of the instructional staff and the program 
director. Through the use of the previously mentioned evaluative 
techniques, it did appear that the principle objectives and goals 
of the Institute v;ere successfully accomplished. 

Program factors . The evaluation of the portions of the institute 
devoted to research design and statistics indicate that the objectives 
were accomplished. The pre-test (patterned after Stanley's AERA 
approach) sampled the level of understanding that the trainees had with 
regard to research terms and concepts. Their understanding and 
computational ability with regard to statistics and research design as 
measured on a five point scale on the pre-test indicated that the 
understanding level and computational skills were predominantly at 
the middle and lower level of the scale. When the same instrument 
was applied as a post test, the level of understanding and computational 
skills moved sharply to the upper level of the scale. A complete 
summary of the before mentioned pre-test and post test may be found 
in Appendix B. An analysis of the individual participant evaluations 
of the research design and statistics portion of the institute also 
indicated that the goals and objectives were met successfully. The 
participants were asked to respond to questions concerning (1) the 
program of study, (2) organization and administration, (3) expectations, 
and (4) to make any additional comments concerning the institute. The 
participants' evaluations of the program of study were positive with 
particular emphasis given to the appropriateness of material presented 
and the high quality of class instruction. The participants also 
indicated that the organization and administration of the institute was 
very satisfactory in all aspects except for not providing housing on the 



campus . 



10 



The evaluation of the one week session on Program Evaluation Review 
Technique met the expectations of the participants and the objectives 
of the institute. All of the participants agreed that the instruction 
and materials received X70uld contribute a great deal to their work in 
educational research and development. A complete summary of the partici- 
pants' evaluation of the session on PERT may be found in Appendix B. 

The participants' reactions to the three days spent on Automatic 
Data Processing was positive. The high quality of instruction and the 
appropriateness of materials covered were the areas most often mentioned 
by the trainees. The visitations to two separate centers using automatic 
data processing was a worthwhile experience for the participants. 

The participants did not find the one day spent on ERIC as worth- 
while as the session on Regional Laboratories and R. and D, Centers. 

The major criticism of the one day spent on ERIC Xi7as based on major 
emphasis being placed on a local operation rather than an overall orienta- 
tion and overview of the program in general. 

The number of staff members was quite adequate for the number 
of participants. With eighteen trainees attending the first session 
and nineteen attending the second session, it was possible for the 
instructional staff to provide more attention to individual needs. This 
smaller group also enhanced the effectiveness of the discussion periods. 
The instructional staff had sufficient time to adequately prepare assign- 
ments, duplicate hand-out materials, and testing material. The director 
and the research assistant were able to schedule sufficient time to 
perform the necessary organizational and administrative tasks. Adequate 
secretarial assistance was provided for all aspects of the institute. 
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The Bureau of Educational Research, University of Denver, was 
not involved in the development of selection criteria for the institute. 

The names of the participants selected for the institute were submitted 
to the institute director from the United States Office of Education, 
Division of State Agency Cooperation and the Division of Research 
Training and Dissemination. The original list of participants nominated 
for the institute consisted of thirty candidates. Twenty-eight candi- 
dates preregistered for the institute. Eighteen candidates attended 
the first two week session while nineteen attended the second two week 
session. Those candidates who preregistered but did not attend, 
indicated that their absence was due to unexpected job responsibilities 
and conflicting dates. A number of candidates were unable to attend 
due to their responsibilities in connection with state legislative 
sessions. 

There did not appear to be any major problems related to the 
organization of the institute. The trainees particularly favored 
dividing the institute into two two-week session spaced approximately 
a month apart. Daily schedules were planned in advance so that train- 
ees had a long-range viex^ of the presentations. Minor modifications 
were made in order to adjust to the rate of progress of the trainees. 

The classroom was adequate for the number of trainees who participated. 

A variety of resource materials was available for the participants through 
the Bureau of Educational Research and the University library. 
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The budget for living allowance, staff salaries and time was more 
than adequate. Several minor internal line item transfers were necessary. 
This was particularly true in the area of supplies-expenses and rental 
of equipment. The initial budget estimate for supplies and expenses 
was not adequate while the estimated budget for rental of equipment 
was considerably more than needed. The total amount expended was $20,810.32 
less than originally budgeted. This was due mainly to the number of 
trainees being less than expected. 

Strengths and weaknesses . The major strength of the institute 
was undoubtedly the instructional staff employed. The professional 
staff employed were unquestionably outstanding authorities in their 
respective fields. Their enthusiasm for teaching and methods of presen- 
tation was reflected in the high motivation of the trainee group. The 
methods of presentation and media used were characterized by a variety 
of procedures. All the participants were unanimous in their 
assessment of the following; 

1. The purpose of the institute was clearly outlined 

2. The objectives were realistic 

3. Materials presented were of value to each individually as 
well as collectively 

4. Solutions to individual problems and questions were 
considered 

5. The instructors were well qualified 

6. The sessions followed a logical order 
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7. The schedule provided for adequate flexibility without loss of 
continuity 

8. Opportunity was provided for exchange of ideas and informed 
discussion 

9. The institute was well organized and administered 

10. The large volume of duplicated materials) texts, and other 
resource materials provided were appropriate and will be of 
value both immediately and in the future. 

The only major weakness encountered in the institute was the 
inability of the university to provide housing on campus for all the 
participants. 

It was the opinion of some of the participants that the time 
spent on ERIC and Regional Laboratories would have been better spent 
on an additional two days on automatic data processing. (This was 

particularly true of the day spent on ERIC.) 

The facilities provided were not unusual, though adequate. The 
timing of the institute was satisfactory except in those cases where 
the state department employees had obligations in legislative matters 
in those states where the legislature was in session during the time 
of the institute. 

No minor difficulties were encountered with the administrative 
relationships with the USOE. The office was very cooperative in 
answering all inquiries. The only suggested improvement x^ould be in 
allowing the project director to approve or disapprove substitute 
enrollees when a late cancellation of a previously approved candidate 



occurs* 
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The overall evaluation of the institute indicates that it was 
successfully carried out, that the trainees benefited considerably, 
and that the institute accomplished its objectives. This statement is 
supported by the pre-tests and post-tests administered, the individual 
written overall evaluation by each trainee, the opinion of the instruc- 
tional staff, the opinion of the project director, and the informal 
observation of comments by the participants. 




PRX)GRAH REPORTS 

1, Publicity 

The institute was publicized through a written communique 
to the chief State School Officers from the U.S.O.E., 

Office of the Director, Division of State Agency Cooperation. 
This communique explained the nature and purposes of the 
Institute, location, schedules, and the name of the director. 
The state school officer had the opportunity to nominate 
three people to participate. Candidates accepted sent 
preregistration forms directly to the Director of the 
Institute. All preregistered nominees were contacted by 
the Director of the Institute and sent information concerning 



housing, travel, classroom location, and a more detailed 
resume of the nature of the curriculum, 

2, Application Summary 

a. Approximate number of inquiries from 
prospective trainees 

b. Number of completed applications 



received 

c. How many applicants were offered 



35 



Done by 
USOE 



33 



admission 
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3, Trainee Summary 

a. Number of trainees initially accepted 
in program 

Number of trainees enrolled at the 

beginning of program 

Number of trainees who completed 

program 

<lai-eg oj?i zat ion of trainees 
^1) Number of trainees who principally 
are •elementary or secondary school 
teachei;s 

(2) Number of trainees who are 
principally public school 
administrators or supervisors 

(3) Number of trainees from state 
education groups 

(4) Number of trainees from colleges 
or universities, junior colleges, 
research bureaus, etc. 

4, Program Director’s Attendance 

a. What was the number of instructional 
days for the program? 

b. What was the percent of days the director 
was present? 



30 



18 



19 



none 



none 



19 



none 
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RESEARCH 


FINANCIAL SUMMARY 
INSTITUTE - PROJECT 


NO. 7-0359 




DIRECT COSTS 


BUDGETED 


EXPENDED 


BALANCE 


Personnel 

Faculty 


$ 4,400.00 


$ 4,400.00 


0 


Secretary 


800.00 


800.00 


0 


Graduate Assistant 


500.00 


500.00 


0 


Fringe Benefits 


495.00 


495.00 


0 


Staff Travel 


1,475.00 


646.30 


828.70 


Staff-Per Diem 


640.00 


720.00 


-80.00 


Supplies and 
Equipment 


1,390.00 


1,110.91 


279.09 


Trainee Support 
Travel 


4,800.00 


4,826.11 


-26.11 


Per Diem 


12,480.00 


7,312.00 


5,168.00 


Totals 


$26,980.00 


$20,810.32 


$ 6,169.68 




APPENDIX A 



PEIE-REGISTRATION FORM 



Research Training Institute for State Agency Personnel Co 
the Division of Research Training and Dissemination and the 
State Agency Cooperation of the United States Office of Education. 



Dear Institute Participant; 

would you take a few moments to fill in the 0®“i*rie'’ft°^ossibirfor' 
to the Institute Director? Your cooperation i„ 

those conducting the Research Institute to gear the content more nearly 

to your background and needs. 

Name: 



Position Title: 



State: 

How long have you been in your current position? 

How long have you been in the State Department? 



Length of service in education? 



What level (s)? Elementary 



Secondary 



College 



Degrees: Bachelors Major 

Masters Major 

Doctorate Major 



What research responsibilities do you have in 



the State Department? 





Have you had courses in statistics? 
If so, how many hours? 

Do you have training in research? __ 
If so describe briefly. 



1 $ 

2 



Are you now responsible for any program of research? 
If so, what kind? 



Please send completed form to attached address. Thank you. 



UNIVERSITY OF DENVER 

COLORADO SEMINARY 



University Park 

Denver, Colorado 80210 



BUREAU OF EDUCATIONAL RESEARCH 

March 15, 1967 



The first meeting scheduled for the Research Institute for State 
Department Personnel will be held at 9?00 A.M, , March 27, 1967, at 
the Bureau of Educational Research, University of Denver. I have 
enclosed a map of the university campus with the Bureau of Educational 
Research marked by a red circle. 

Living accommodations have been arranged for you at the Belcaro 
Motel. This motel is approximately two miles from the campus. Arrange- 
ments have been made to provide transportation for you from the motel 
to the canpus on Monday, March 27. Please be at the Belcaro office at 
8:45 A.M, , Monday. Should you plan to arrive later than 6:00 P.M. on 
either March 25 or March 26, you should personally contact the Belcaro 
Motel. I have also enclosed a brochure from the motel. 

Should you have any problems or questions when you arrive in 
Denver, please call Mr. Lee Thomson, home phone number 757-1337, or 
Mr. Alvie Shaw, home phone number 757-2508. 

If you are traveling by public carriage, please retain your ticket 
receipt. We will make every effort to reimburse your travel expenses 
at the end of the first week of the institute. 

Other necessary information will be provided at the first session 
of the institute. 

We are looking forward to seeing you in the near future. 

Sincerely, 



Alvie L. Shaw 

Assistant to Dr. R, A, Fons.ythe, 
Institute Director 



ALSrbaj 



Enclosures: 2 






i- 

[ 
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RESEARCH INSTITUTE 
FOR 

STATE EDUCATIONAL AGENCY PERSONNEL 
Session I ~ March 27, 1967, to Aptll 7, 1967 
Measurement in Educational Research, Research Design, and Program Evaluation 

AGENDA FOR OPENING SESSION 
Monday - March 27, 1967 



9:00 A.M. 
9:30 A.M. 

10:00 A.M. 

10:15 A.M. 
10:30 A.M. 



-9:30 A.M. 

- 10:00 A.M. 

- 10:15 A.M. 

- 10:30 A.M. 

- 11:00 A.M. 



Informal Get-Acqualnted Coffee - Conference Room - 
Bureau of Educational Research, University of Denver 

Welcome and Introductory Talks 

1. Dr. R. A. Forsythe - Institute Director 

2. Dr. E. A. Llndell, Dean, College of Arts 

and Sciences 

3. Dr. W. M. Slalchert, School of Education 

Information Concerning Details of Institute 
(housing, travel reimbursement, meals, lab sessions, 
etc.) 

Mr. Alvle Shaw - Research Assistant 

General Overview and Format of Institute 

Dr. R. A. Forsythe - Institute Director 

Outline of Course Work Content and Course Work 
Procedures 

Dr. Kenneth Hopkins - Instructor 
Dr. Gene Glass - Instructor 



11:00 A.M. - 11:30 A.M. 


Pre-test Given to All Participants 

(Measurement In Educational Research & Research Design) 


11:30 A.M. - 1:00 P.M. 


Ldnch Break 


1:00 P.M. - 2:20 P.M. 


Dr. Hopkins (Lesson I) 


2:20 P.M. - 2:40 P.M. 


Coffee Break 


2:40 P.M. - 4:00 P.M. 


Dr. Glass (Lesson II) 




Park 



I UNIVERSITY OF DENVER University 

I 

t COLORADO SEMINARY Denver, Colorado 80210 

t BUREAU OF EDUCATIONAL RESEARCH 

April 18, 1967 



The first meeting scheduled for the second half of the 
Institute (May 1 through May 12) for State Department Personnel 
will be held at 9:00 A.M. , May 1, 1967, at the Bureau of 
Educational Research, University of Denver. Enclosed you will 
find a complete schedule for the first week of the institute. 

If for some reason you find you are unable to attend this 
two week session, I would appreciate your notifying me by return 
mail . 



Should you have any problems or questions when you arrive 
in Denver, please call Mr. Lee Thomson, home phone number 
757'-1337, or Mr. Alvie Shaw, home phone number 757-2508. 

We are looking forward to seeing you on May 1, 1967. 

Sincerely, 



Alvie L. Shaw 

Assistant to Dr. R. A. Forsythe 
Institute Director 



ALS :baj 
Enclosure 






23 



BUREAU OF EDUCATIONAL RESEARCH 

University of Denver 
Denver, Colorado 

INSTITUT E 

Second Two Week Session (May 1 through May 12) 



Program 


Date 


Instructor 


P.E.R.T. 


May 1 through May 5 


Desmond Cook 
Mrs. King 



Automatic Data 


May 8, 9, 10 


Ralph VanDusseldorp 


Processing 


Pete McGraw 


E.R.l.C. 


May 11 


Edgar Charles 


Regional Educational 


May 12 


James Thrasher 



Laboratory and R.D. 
Centers 
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SEOONO 1HO HUX SESSION 

The yrtanlgatlpn 

The second session «£ the Institute progrem will be conducted 
es e workshop. The workshop will consist of lectures t discussions , 
labpretory experience • and visitations.. The ten day session will 
be scheduled as follows: 



P.6.R.T. 


5 days 


Hay 1 to May S 


Automatic Data 


3 days 


May 8, 9, 10 


Proeesglng 






E.k.I.Ci 


1 day 


May 11 


RcgionSl Educational 


1 day 


May 12 



Labosatoty 



A «ore detailed btesWown of the four areas to be covered isf 
Automatic Data PrdeessinK 

1. OrientAtion 3. Utilisation 

a. dstds s. cards analysis 

b. dorters b« computer analysis 

2. Computer 

a* punching and programming 
b» iystemA (schools) 

(The program will include laboratory experiences with the various 
data proceadibg e<iuipment|^ There will also be arranged visitations 
to the InteknAtional Business Hschines Training Center In Denver, 
The DeMier txmtitutai s schoc! district utilising data 



o 



I 

1 



processing, and the Colorado State Department of Education data 
processing center.) 



Program Evaluation and Review TechniauA 



1. General Orientation, 

2. Network Concept, 

3. Flow Chart Techniques, 

Application of Network Concepts and Flow Chart Techniques. 




1. Lecture, display of materials, and general discussion. 
Regional Educational Laboratorfskfl 



1. Lecture and general discussion, 

2. Visit to the Rocky Mountain Educational Laboratory. 



o 

ERIC 



RESEARCH INSTITUTE 
FOR 

STATE EDUCATIONAL AGENCY PERSONNEL 
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Name 

1. Jerry Barton 

2. Milt Baum 

3. Charles Bostrom 

4. Duane Carr 

5. Elmer Clausen 

6. Joanne Clenmer 

7. Richard Gunkel 

8. William Hib^ar 

9. Perry Keithley II 

10. Lamar LeF.evre 

11. James McNamara 

12. Robert Nichols 

13. Charles Nix 

14. Elvin H. Ossmen 

15. Chris Pipho 

16. Francis Rlst 



Bureau of Educational Research 
University of Denver 
Denver, Colorado 80210 

PARTICIPANTS 

Position 

Director of Research 

Director of Research 

Director, Division of Statistics 
and Data Processing 

Statistical Analyst 

Director, Adult Education 

Research Analyst 

Director of Federal Programs 

Program Assistant, Safety Education 

Statistician 

Coordinator, 8 State Project 
Research Associate 
Occupational Research 
Director, Assessment and Research 
Specialist, Research and Statistics 
Consultant, Research and Development 
Teacher Education and Certification 




Location 

Texas 

Oregon 

Colorado 

Utah 

Washington 

Oregon 

Nevada 

Washington 

Washington 

Nevada 

Pennsylvania 

Texas 

Texas 

Utah 

Colorado 

Idaho 
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Name 




Position 


Location 


17. 


Billy Siler 


Director of Research 


Oklahoma 


18. 


Howard Snort land 


Director, Finance Statistics 


North Dakota 


19. 


Quentin Utley 


Administrator, Division of Elementary 
and Secondary Education 


Utah 


20. 


Gustave Lieske 


Director of Research 


Nebraska 


21. 


David Jesser 


Project Director 


Nevada 









o 

ERIC 

hiaifiiifftaiTiaaa 



APPENDIX B 



o 

ERIC 

hiaifiiifftaiTiaaa 



29 



DR. HOPKINS 

MondaVt A.M. « March 27, 1967 
Role of research and education 
Basic statistical concepts 
Measure of variability 
Skewness 

Normal distribution 



Tuesday. A. M. « March 28, 1967 
Research Resources 

Encyclopedia of Educational Research 
Handbook of Research on Teaching 
Review of Educational Research 
Mental Measurements Yearbook 
Tests in Print 

Selecting a Criterion 

’’Jingle” and ’’jangle” fallacies 

Methods of Reporting Results -- Comparing Relative Performance 



Norms and Reference Groups 

Inadequacy as a control group 
Importance of time of administration 
Extrapolation-Interpolation of units 

Grade Equivalents 

Plateau effects 

Non- equivalence within a battery or between tests 



Percentiles 

Inequality of units 

Standard Scores 

Advantages 
Z- scores, T-scores 
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DR. HOPKINS 



Wednesday. A.M. « March 29, 1967 
Correlation, Regression and Prediction 
Interpretation 

Not a ’’per cent", inequality of units 
Not causation 

Independent of mean and variability differences between 
X and Y 

Relative distance from mean (Zy^ = ^Zx) 

Effects of heterogeneity- selection 

Prediction 

Standard error of estimate 
Education Illustrations 

Driver training, TMR, ERM programs 



Thursday. A. M. . March 30, 1967 
Expectancy 

Statistical Significance 

Sampling distribution 
Standard error 
.05, .01 levels 
vs. practical significance 

Confidence Intervals 

Type I and II errors 



o 
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DR. HOPKINS 






FridaVt A. M. . March 31, 1967 
Comparing means - t-ratio 

Interpreting statistically significant difference 
Interpreting confidence intervals 

Measurement of "Intelligence”, scholastic aptitude, etc. 

Multi-dimensional nature 
Meanings of IQ score 



Group vs. Individual tests 

Language-Verbal test vs. Non- language Performance test 

Ratio vs. Deviation IQs 

"Cultures-fair" tests 

IQ constancy and age -- role of chance 

Reading factor and IQ scores 

Nature-nurture considerations 

Prediction vs. Potential 

Standard error of measurement 
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DAILY SCHEDULE 
MEASUREMENT - STATISTICS 
DR. HOPKINS 

Monday. A. M. , April 3, 1967 
Differences in propostion 
Chi square 

Sampling 

Confidence interval for proportions 
Sample size effects 

The analysis of variance 

Interpretation and utilization 

Two factor designs 

Role in '’individualized'* instruction 
Interaction 

Practice in interpretation 
Tuesday. A.M. . April 4, 1967 

Review of Basic Concepts in Statistical Inference 

Further consideration of factorial designs 

Three factor designs 
Two factor interaction 

Rationale and interpretation 
Wednesday. A. M.. April 5, 1967 

Further considerations of factorial design and interaction case studies 

The analysis of covariance and quasi-experimentation 

Role in compensation for bias- adjusted means 
Role in statistical power consideration 
Assumptions 

Illustration and interpretation 



o 
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PR. HOPKINS 

Thursday. A. M. . April 6. 1967 
Additional illustrations of ANCOVA 
Interpretation of higher order interactions 

Lab session: designing a research study to answer specific questions 

Friday. A. M. . April 7, 1967 

Measurement of Affective Variables 

General problems 

Self-report documents 
Opinionaire and questionnaire 

Newer technique 

Semantic Differential 
Q-sort 

Social Measurement - resources 
Anxiety measurement 
Creativity 



o 
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DAILY SCHEDULE 
Research Design - Dr. Glass 
Monday. P. M. . March 27, 196? 

1, Pre- experimental and quasi-experimental design 

2. Sources c2 internal invalidity of experiments 
Materials handed out: 

1. "Definition of Sources of Internal and External Validity" (Glass) 

2. "A Critique of Experiments on the Role of Neurological Organization in 

Reading Performance" (Glass) 

3. "Experimental and Quasi-experimental Designs for Research" 

(D. T. Campbell and J* C. Stanley) 

Tuesday. P. M. . March 28. 1967 

1. Sources of external invalidity 

2. Discussion of illustrations of internal and external invalidity 
Materials handed out: 

1. "Illustrations of Internal and External Invalidity" (Glass) 

Wednesday. P. M. . March 29. 1967 

1. Discussion of studies in "A Critique of Experiments on the Role of 
Neurological Organization in Reading Performance" 

a. . The internal and external validity of selected experiments 

b. Causation and correlation 

Thursday. P. M. March 30. 1967 

1. True experimental designs and quasi-experimental designs 

2. Experimental unit vs. unit of analysis 

3. Mastery test on experimental design 
Materials handed out: 

1. "The Experimental Unit and the Unit of Statistical Analysis: 

Comparative Experiments with Intact Groups" (Glass) 

2. "The Countenance on Educational Evaluation" (R. E. Stake) 
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DR. GLASS 

Friday. P. M.. March 31. 1967 

1. Common misconceptions about evaluation 

2. Introduction to a Modern Strategy of Educational Evaluation 
Materials handed out: 

1. "Remarks on Evaluation of Education Programs" (Glass) 

2. "Misconceptions about Evaluation? 
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DR. GLASS 

Monday. P.. M. . April 3. 1967 

1. The elements, operations and activities of the evaluation of 
educational programs 

2. Types of evaluation activities: formative vs. summative; 

instrumental vs. consequential 

3. Discussion of the nature of evaluation plans in a sample of 
4$4 Title I proposals 

Materials used: 

1. "Materials on the Evaluation of Educational Programs" (Glass), pp. 1-2 

2. "Types of Evaluation Techniques in Title I Project Proposals" (Morrison) 

Tuesday. P. M. , April 4. 1967 

1. The plan for an evaluation of a fictitious Title I program 

2. The assessment of attitudes 
Materials used: 

1. "Remarks on the Evaluation of Educational Programs" (Glass), pp. 3-7. 
Wednesday. P. M. . April 5. 1967 

1. Examination of an exemplary evaluation of an educational program 

2. Discussion of bibliography of references to other exemplary 
evaluation reports 

Materials used: 

1. "Report of the AERA 1966 Pre-session on Experimental Design" 

(Stake, Glass, Taylor) 

2. "Examples of Evaluations of Educational Programs" (A bibliography) 
Thursday. P. M.. April 6. 1967 

1. Distribution and discussion of a bibliography of 230 references 
in educational evaluation 

Materials used: 

1. "Bibliography of References in Evaluation" (Stake) 
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Friday. A.M. . April 7. 1967 

1. Administration and scoring of a mastery test on evaluation 
Materials used: 

1. "Mastery Test on E%^aluation" (Glass) 
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Bureau of Educational Research 
University of Denver 
Denver, Colorado 80210 



Session II - May 1, 1967, to May 5, 1967 



DAY 


TOPIC 


INSTRUCTOR 


REFERENCE 


Monday - AM 


Administrative Matters 
Orientation 
Nature of Management 
Management Process 
Management Information 
Systems 

Research Management 
PERT History 


Cook 


Cook, pp. 1-9 
Cook - "New Approach' 
Woodgate, Ch. 1 


Monday - PM 


Establishing Information Base 
Introduction to PERT 
Network Construction 


King 


Cook, pp. 10-19 
Woodgate, Ch. 2,3,4 
PERT Film 


Tuesday - ^ 


Establishing the Time Base 
Activity Time Estimation 
Network Analysis 


King 


Cooic, pp. 19-31 
Woodgate, Ch. 5-6 


Tuesday - ^ 


Work Session 


King and Cook 




Wednesday - M 


Scheduling the Project 
Resource Allocation 


Cook 


Woodgate, Ch. 8,12 


Wednesday - 


Establishing the Cost Base 

PERT/Cost 

Program Budgeting 


King 


Cook, pp. 31-34 
Woodgate, Ch. 10,11 
PERT/Cost film 




Computer Processing of 
Base Data 


Cook 


Cook, pp. 72-76 
Woodgate, Ch. 14 


Thursday - ^ 


Controlling as a 
Management Function 
Up-dating 
Management Reports 
Problem Identification 
Decision-making 


Cook 


Cook, pp. 77-83 
Woodgate, Ch. 9 


Thursday « PM 


Group Discussion 
on Application of 
Management Information 
Systems to Education 


Cook, King 
and Group 
Leaders 


Cook, Ch. 3 



Friday - AM 



Educational Applications 
PERT Implementation 
Summary 
Critique 



Cook 

King 

Cook 



Cook, pp» 83*'86 
Woodgate, Ch. 13 
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DAY 


TOPIC 


INSTRUCTOR 


Monday - M 


Introduction to Data Processing in Education 
Administrative Uses 
Teaching A. Df P. 

Computer Assisted Instruction 


McGraw 




Introduction to Electro-Mechanical Machines 
The Punched Card 
Key Punch 
Verifier 
Sorter 

Accounting Machine 


McGraw 


Monday - PM 


Introduction to Electro-Mechanical Machines 
(Continued) 

Interpreter 

Reproducer 

Collator 


VanDusseldorp 




Introduction to Computers 


VanDusseldorp 


Tuesday - M 


Demonstration of Electro-Mechanical Machines 


McGraw and 
VanDusseldorp 




Computers (Continued) 


VanDusseldorp 


Tuesday - M 


Computers (Continued) 


VanDusseldorp 




Computer Programming 
Systems Development 
Computer Demonstration 


VanDusseldorp 


Wednesday - 


Research Applications 


VanDusseldorp 


Wednesday - M 


Local School Relations 


VanDusseldorp 




APPENDIX C 




41 



CRITIQUE OF INSTITUTE BY PROFESSIONAL STAFF 

FIRST SESSION 
Dr. Gene Glass 



Indicate your observation and judgment by checking each item in one 
column at the left, then by amplifying your response in ‘he blank 
at the right when appropriate. Use additional paper . 

not applicable or not subject to your observation should be omitted. 

Be frank. 



1. Environmental conditions 

a. Classroom spaces 

b. Work spaces 

c. Living quarters 

d. Teaching equipment, aids (chalk boards, 
public address system, etc.) 

e. Resource material, library 

f. Eating facilities 



Setting was more 
uncomfortable than 
need be. Chairs were 
murder. Noise was 
bad. 



2. Participants 

a. Appropriateness of academic backgrounds 

b. Sufficiency of research experience 

c. Willingness to work 

d. Intellectual curiosity 

e. Concern for applicability of techniques 

f. Aspiration 

g. Immediate preparation for Presession 



3. Organization 

a. Adequacy of notice to prospective ) 

applicants I 

b. Sufficiency of preplanning \ 

c. Smoothness of opei^ation [ 

d. Adaptability to obstacles and \ 

feedback 

e. Sensitivity to grievances ^ 

f. Adequacy of financial support 



Not in a position 
to judge. 



4 . Schedule 

a. Appropriateness of 10 days for the job 

b. Time spent efficiently 

c. Events sequenced appropriately 

d. Punctuality 

e. Balance between formal, informal affairs 

f. Quantity of discussions 

g. Quality of discussions 

h. Quality of formal presentations 

i. Unobtrusiveness of evaluation efforts 

j. Methods of evaluation 



Overly long, perhaps. 



Provide more oppor- 
tunities for informal 
discussions . 



Quite good, really. 
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5. Outcomes 

a. Intended content was actually taught 

b. Increase in participant understanding . 

c. Improvement in attitude toward research Hard to say._ Hope- 

d. Personal associations initiated fully satisfactory* 



6. In general was the Presession well organized? 

Yes, though preparation time for myself was too short. 

Construction work going on outside 104 Iliff was annoying and 
distracting. If it*s still going on in May, change lecture 

rooms . 

7. Were the facilities at the Center (not the motel) suitable for the 
activities which you had planned? If not, specify. 

Not applicable. 

8. Did it make a difference, favorable or unfavorable, to have the Presession 
scheduled in the same city as the AERA meetings? 

Not applicable. 

Should Presessions be limited to the same hotel, or the same city, in which 
the annual meetings will be held? 

Not applicable. 

9. Were you satisfied with the group of "student" participants selected? 

How could the selection have been improved? 

Participants appear to have been "tapped" by someone in their office 
as opposed to having applied of their own volition. 

10. Did you perceive the participants to be reasonable well satisfied with the 
10 day experience? 

By and large. 

11. As a lecturer were you bothered by interruptions of your lectures? 



No. 



12. At two points during the Presession sets of questions about the organization 
and management of activities were administered to the participants. Their 
responses were tallied and given to you. How did you respond to the know- 
ledge of the participants' responses? Did you disregard them? Change 
plans? Did you find this polling of participants useful or nearly useless. 

The pre-test was quite useful to me in lecturing. 
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13. Were you to do the same as.slgiunent over, in what major ways, if any, would 
you change your contribution? 

14. Do you wish that the Director had made firmer arrangements to assure 
participants and you of the staff opportunity to meet in pairs or small 
groups? 

Yes. 

15; Were the objectives you set for yourself during the Presession attained? 
(Have we missed opportunities to evaluate how well objectives have been 
attained?) 

< Objectives were attained by and large. 

16. Are you inclined to urge your colleagues to become staff members for such an 
institute ot Pr^seaslon? 

Depends on which colleague. 

17. Do you believe that the research practices of the participant group will be 
improved as a result of their participation? 

Difficult to say. They should become better (more critical) 

consumers and evaluators of research. 

18. In what ways, if any, did you as a staff member benefit personally as a 
result of your participation in this Presession? 

Gained familiarity with a formerly unknown element (State Department)* 

19. In your opinion, how does education benefit from this kind of session? 

It most certainly benefits. 
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CRITIQUE OF INSTITUTE BY PROFESSIONAL STAFF 

FIRST SESSION 
Dr. Kenneth Hopkins 



Indicate your observation and judgment by checking each item in one 
column at the left, then by amplifying your response in the blank at 
the right when appropriate. Use additional paper if needed. Items 
not applicable or not subject to your observation should be omitted. 

Be frank. 

1. Environmental conditions 

a. Classroom spaces 

b . Work spaces 

c. Living quarters 

d. Teaching equipment, aids (chalk boards, 
public address system, etc.) 

e. Resource material, library 

f. Eating facilities 

2. Participants 

a. Appropriateness of academic backgrounds 

b. Sufficiency of research experience 

c. Willingness to work 

d. Intellectual curiosity 

e. Concern for applicability of techniques 

f. Aspiration 

g. Immediate preparation for Pre session 

3. Organization 

a. Adequacy of notice to prospective 
applicants 

b. Sufficiency of preplanning 

c. Smoothness of operation 

d. Adaptability to obstacles and feedback 

e. Sensitivity to grievances 

f. Adequacy of financial support 

4. Schedule 

a. Appropriateness of 10 days for the job 

b. Time spent efficiently 

c. Events sequenced appropriately 

d. Punctuality 

e. Balance between formal, informal affairs 

f. Quantity of discussions 

g. Quality of discussions 

h. Quality of formal presentations 

i. Unobtrusiveness of evaluation efforts 

j. Methods of evaluation 
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5, Outcomes 

a. Intended content was actually taught 

b. Increase in participant understanding 

c. Improvement in attitude toward research 

d. Personal associations initiated 



6. In general was the Presession well organized? 

Yes« 

7. Were the facilities at D.U. suitable for the activities which you had 
planned? If not, specify. 

Yes. 

8. Did it make a difference, favorable or unfavorable, to have the 
Fresession scheduled in the same city as the AERA meetings? 

Not applicable. 

9. Were you satisfied with the group of ’’student” participants selected? 

Yes. 

How could the selection have been improved? 

Better description of program to participants and State Department 
of Education. 

10. Did you perceive the participants to be reasonably well satisfied with 
the 10 day experience? 

Yes. 

11. As a lecturer were you bothered by interruptions of your lectures? 

No. 

12. At two points during the Presession sets of questions about the organiza- 
tion and management of activities were administered to the participants. 
Their responses were tallied and given to you. How did you respond to 
the knowledge of the participants* responses? Did you disregard them? 
Change plans? Did you find this polling of participants useful or nearly 
useless? 

Did not disregard them. Changed plans. Found the polling of 
the participants to be nearly of the same value. 
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13. Were you to do the same assignment over, in what major ways, if any, would 
you change your contribution? 

No major changes. 

14, Do you wish that the Director had made firmer arrangements to assure 
participants and you of the staff opportunity to meet in pairs or 
small groups? 



No. 

15. Were the objectives you set for yourself during the Presession attained? 
(Have we missed opportunities to evaluate how well objectives have been 
attained?) 

I think the objectives were attained. 

16. Are you inclined to urge your colleagues to become staff members for 
such an institute or Presession? 

Yes. 

17. Do you believe that the research practices of the participant group will 
be improved as a result of their participation? 

Yes. 

18. In what ways, if any, did you as a staff member benefit personally as 
a result of your participation in this Presession? 

From the feedback from the "field." 

19. In your opinion, how does one benefit from this kind of Presession? 

Not applicable. 



I judge the two week session to have been highly successful in 
attaining its objectives. The group was very responsive and appear- 
ed to follow through very well on assignments. Several of the 
participants expressed regret that some of their colleaguas were 
not here, and felt they might have been if they had known more 
explicitly the nature of the institute. A couple felt that some 
useful information would have resulted if they had united 
together as a group- -they would be resource persons for each 
other. In total, however, this institute was more successful 
than I really had anticipated. 
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INSTITUTE PARTICIPANT EVALUATIONS 
Please respond briefly to the following questions: 

I. Regarding program of study. 

A. What were the strengths of the Institute program? 

A very good staff -- and course content. 

B. What were the weaknesses of the Institute program? 
ERIC 

C. How appropriate were the materials covered for your 
particular needs? 

Very good 

D. How would you improve the program? 

Needs an extension in the area of evaluation. 

II. Regarding organization and administration. 

A. Was the length of the Institute adequate? 

Yes 

B. Was the time of year satisfactory? 

Yes 

C. Were housing accommodations satisfactory? 

Yes 

D. Were classroom accommodations satisfactory? 

Need better classrooms 

E. Did you have adequate materials? 

Yes 

F. Were you satisfied with the manner by which the 
administrative details were carried out? 

Yes 
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III. Expectations 

A. What did you expect to get out of the Institute before 
you came? 

Very much the same as presented, 

B, Were your expectations reached? 

Yes 

IV, Please list any other comments you wish to make concerning the 
Institute, (First and/or second session) 

It was A well run institute^ Exceptional staff . 



o 
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INSTITUTE PARTICIPANT EVALUATIONS 



Please respond briefly to the folic questions: 

I, Regarding program of study. 

A. What were the strengths of the Institute program? 



The quality of the instructors. 

B. What were the weaknesses of the Institute program? 



Classroom facilities; inconvenient housing ^ 
*Insufficient togetherness as a group in eating, housing, 



entertainment . 



C. How appropriate were the materials covered for your 
particular needs? 



Very good 

D. How would you improve the program? 



See above. 

A session on staffing and operating 
development program. 



a local or state research and 



II. Regarding organization and administration. 

A. Was the length of the Institute adequate. 

I don't think it could be shortened but my duties at home 
suffered because it was too long. 



B. Was the time of year satisfactory 

The first two weeks interferred with legislative sessions. 

C. Were housing accommodations satisfactory? 

NO 

D. Were classroom accommodations satisfactory? 

NO 

E. Did you have adequate materials? 

The second two weeks— NOT the first txro weeks. 



o 
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III. 



IV. 



ERJC 



F. Were you satisfied with the manner by which the 
administrative details were carried out? 

YES 

Expectations* 

A, What did you expect to get out of the Institute before 
you came? 

Something more than I did in regard to staffing and operation of 
a local or state department of research and development* 

B. Were your expectations reached? 

Kora than sufficient except for item III A. above. 

Please list any* other comments you wish to make concerning the 
Institute* (First and/or second session) 

We are ever so grateful to instructors; the Bureau administration; 
the graduate assistants, and the office women. 
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INSTITUTE PARTICIPANT EVALUATIONS 
Please respond briefly to the following questions: 

I. Regarding program of study, 

A. What were the strengths of the Institute Program? 
Knowledge of instructors and effective time parameter. 

B. What were the weaknesses of the Institute progi^ m? 
ERIC -- not necessary 

C. How appropriate were the materials covered for your 
particular needs? 

Excellent 

D. How would you improve the program? 

No comment 

II. Regarding organization and administration. 

A. Was the length of the Institute adequate? 

Yes 

B. Was the time of year satisfactory? 

Yes 

C. Were housing accommodations satisfactory? 

D. Were classroom accommodations satisfactory? 
Questionable noise, etc. 

E. Did you have adequate materials? 

Yes 

F. Were you satisfied with the manner by which the 
administrative details were carried out? 

Yes 
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III. Expectations. 

A. What did you expect to get out of the Institute before 
you came? 

Knowledge in content and identification of personal materials 
in research, statistics, and data processing. 

B. Were your expectations reached? 

Very definitely, 

IV. Please list any other comments you wish to make concerning the 
Institute. (First and/or second session) 

This by far was one of the best workshops I have ever attended. 
Particularly in regard to my responsibilities in Statistics, 

Data Processing, and information system, administratively for 
planning and decision making. 

The staff should be .lommended for their excellence. 
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INSTITUTE PARTICIFAWT EVALUAtlONS 

Please respond briefly to the folloxjing questions: 

I, Regarding program of study. 

A. What were the strengths of the Institute program? 

Administration. 

Excellent instructors. 

B. What were the weaknesses of the Institute program? 

None observed. 

C. How appropriate were the materials covered for your particular 
needs? 

I would have liked more time in Data Processing. 

D. How would you improve the program? 

For over-all needs, I have no suggestion. 

II. Regarding organization and administration. 

A. Was the length of the Institute adequate? 

Yes 

B. Was the time of year satisfactory? 

Yes 

C. Were housing accommodations satisfactory? 

Yes 

D. Were classroom accommodations satisfactory? 

Yes 

E. Did you have adequate materials? 

Yes 

F. Were you satisfied with the manner by which the administrative 
details were carried out? 

Very much so. Hr. Shaw has my congratulations along with the 
rest of the research staff. 
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III, Expectations. 

A, t'Jhat did you expect to get out of the Institute before 
you came? 

I had no particular anticipation. 

B. Were your expectations reached? 

Four X 7 eeks well spent, particularly the second two. 



PlQ33g list any other comments you wish to make concerning 
the Institute. (First and/or second session) 



I could list many; they would all be good. 
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PRE - TEST SUMMARY 

EVALUATION INSTRUMENT 
EDUCATIONAL RESEARCH AND STATISTICS 

Please encircle the code letter which best describes your knowledge 
of each item in the instrument. 

A - Thoroughly understand 
B - Fair understanding 

C - Soma understanding 

D - Have only beard of it 

E - Have never heard of it 

Understanding Level 



B 



1. incernai ana extemcix. vctj.a.uxu> ** *. w , . . — 

2. Ex Dost facto research A BCD E 








7 • 


6 


3- Control group A B C D ^ _ 


2 


11 


4 


1 




4. Random sampling A B C D E 


3 


9 


6 






5- Samoling errors A B c D E 




4 


8 


5 


1 


6- Type I errors A BCD E 


1 


1 


3 


5 


7 


7 . Type II errors A B C D E 




2 


3 


5 


7 


8. Dependent variable ABODE 




4 


6 


4 


3 


9- Indeoendent variable ABODE 


2 


4 


5 


4 


3 


10. Interval scale A B CPE ^ 


1 


1 


6 


4 


5 


11. Nominal scale ABODE 


1 


1 


5 


6 


5 


12. Non- parametric A BCD E^. 


1 


1 


4 


5 


7 


Ti: Population abode T 


5 


7 


5 


1 




14. Continuous vs. discrete data _ A B C D^E — ^ 


1 


2 


5 


7 


3 


15. ”True” experiment ABODE 






6 


6 


is 


16. Normal curve ABODE 


4 


11 


3 






*17. Dichotomized variable ABODE 


1 


2 ’ 


3 


3 , 


9 


1 8 . Quas i-experiment A B_C D E — 






2 


7 


9 


19. Comparative studies A B c D e 


1 


' 1 


13 




1 






4 


12 


1 


1 1 , , , -T 


i21. Hypotheses A B C D E — 


2 


10 


3 


3 




l 22. Null hypotheses ABODE 


2 


5 


4 


4 . 


3 


23. Time series A B C D E __ 


1 




6 


5 


5 


24. Relevance A B C D E_ — ^ 






10 


4 


4 


25. Probability A B C D E 




4 


12 


2 




26. Statistical vs. practical significance A B C D E 




3 


9 


2 


4 


27. Homogeneity of variance A B C D_E 




1 


8 


3 


6 


28':'' Operational defini^^^^^ ABODE 


1 


1 


5 


,7 


4 


29. 0 sort ABC D_E 




1 


4 


4 


9 


30. Experimental unit A B C _D_E ; 


1 5 !. 5 


-1-^ 
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A - Thoroughly understou.'; 

B - Fair understanding 

C - Some understand; ug 

V - Have only heard cf. i‘ 

i5 - Have never heard oi i.t 



9 

J m 



1 . 


Regression effect 


A B 


C D E 


JU • 

2. 


Factorial designs . .. 




*"■ F r 


3. 


__^nrn»2.riate G.vror term 


A 1- 


i‘) E 


4* 


Semantic dlffein^ntirJ. 


7—130-, 


•T •f* *p 

r"-.. V— — - ■ 


5. 


Descr ipr.ive/inf erent ial statistic.3 _ 




'' E 



Und e?' -i i' n r d i n g Level 



A B 



D 



E 



n 



O V* 

)S. 



•h'- V 



n 






-4- 

-I- 



F requ ency di stribution 



"Rgc t an^^ \ar d 1 s t r ib ut io n 
P>,andomlzed blocks 
Statistical model 



Buro ^ s Mental Measurement Yearboolc 



Affe ctiv e /cognitive objectives 
"Rep VO s c ;it f. rive s a mple 

"Tre atment effects 

S taten^e nt of a research problem 
"Sy s t ema t ic bias 

Action research 

7. Finding s vs. conclus ions 

Forcra u. 

Matched group.s 




Collato r 

Table o f r andom numbers 

>5 , F i;:e d - r a ndom-mixed models 

» ■' » " '■* 

Cause and effect 



Rating errors 

^ ^ ^ 



"s t r ong vs. weak a ssumptions 
BGl?ln.it 9r ^n 
Conf oundmg dcuign 



o 
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A 

B 

C 

r> 

E 



Thofoughly understand 
Fail understanding 
Some understand irtf; 
Have only heard of it 
Have never heard of it 



r the following items, encircle the code letter which best describes your 
owledge of the items ^ indicate whether or not you can do the 
mputation by encircling "yes" or "no." 
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POST - test summary 



evaluation instrument 
educational research and statistics 



Please encircle.the code' letter which 
each item in the instrument. . 



best 



describes your knowledge of 



A - Thoroughly understand 
B - Fair understanding 

C - Some understanding 

D *" Have only heard of it 

E - Have never heard of it 



Understanding Level 
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2 



A - Thoroughly undcrnJ*>?rcl 

B - Fair understanding 

C - Some unaerstandiiv, 

' - Have only heard oi . : 

E - Have never heard of it. 

Understanding Level 

JV_ 

-^1 Repression effect 


B C D E 






:2_|_^-_, . L 




32. FacLuti^ii _deb4,giib___ — ^ 


2 10 ! 3 


2 


33. Appronriate erroX-i^Jil — 

OA r1 4 •F Fp Y*Pn *1 1 ^ ^ 


-M- 


J U-- 


1 


jA. b e tr -in L 3 .c_ ^ uj. j. i. u .l u uu j. — ~ — .. . 

r>pcprintiv'’/inferential statistics A_ B,_ 0 _D . b 


6 ,L 




2 ’ 


•a/; T7Y-nmir>n^v d"? S tr ibut ion I , A_..B_^_P„F_ ^ 


9 


1 

4 


S 

1 

) 






A 


6 


4 


37. Rectcnnuler distribution — — — — -i- 

3B Ranconii.?nd blocks A— » 


3 

6 


12 J 


1 

o 


J 


CO model A h C p K ‘ . 


C 


J 


AO Pi,.-«'e T.:'>n<-r»l Tlpasuretiient Yearbook A jj o D E_ . 


1 , 


6 




o 


2 ^. 


Ai A-p-'^rti^i-T-'ro/fopnltive obiectives A 3 1 


1 


4 


7 


r* 

J 


A2 ivc sample A_B..,t< D F . — 


2 


11 


4 






/t^ P.ffoCtG 


1 J, 


10 


^ 7__ 






• 


AA ^t^tpriGnt of T-pnon-roh problGm ^ B C^P E 


1 


4 . 


2 


Aq S\r«?femr.t •? r bias ^ 


1 


7 


6 


3 




- 


li(k Artnnn research A-3- P, ^ ^ ■ - 1 


1 


6 .. 


7 


J-- 


.X-. 


A7 Findinr^n vc5 • conclusions A B C D^E 




P 

U 


6 


3 




Aft ] 


-— 


1 


4 


10 


2 




1 


14 


2 






. i-iai: c U , U U[£D — p p j 

R n TT PT*t t ho *r n P A f f G C t _ — hi — — 


2 !10_ 


2 


3 








4 . 


9 


3 


1 


CO 77* •? r\ T I'l P**r'*i Y * *1 t'^P T7 — * 


1 


5 


10 ... 


2 


*..*• - 1 --— 111 - 


■Rncjplnno dntvi ■ 


u_ 


11 


3 


1 2 






q/i xr-^f r; ru ^ Q__P_J^ 


\J 


9 


2 


2 






I'-rA^C'- j:Ji. — T-., p p. ,7 

cc Wtmn 4 -Vio<*n p?il ronstruct •* V_J-' M ■.-.■■ 


1 




11 


3 


3 




qfi RTMD _A_.B _C_P_^. 


1 

* 




3 


5 


9 


T 7 70y0 . A_B C D E 




2 


5 


5 


5_ 

•n 


Rfl r»b70^ i dccic ^ A J 




1 


4 


5 


,_7__ 


ji. 


qo Nested factors “ - ^ ■—■ ^ •'- ■■ 


; 




3 


7 


6 

Jim 


C\6. Ronlication A 13 C D 


\— 


3 


4 


5 


3 




61- Orthogonal designs - ^ ^ ^ 


\ 


2 


2 


6 

r\ 


7 


69 . Hard sorter 3 B L U t; 


2 


6 


7 


2 




-• 


6X Co" 1 at or ^ B _C P_E ^ — 


-i-A- 


6 


5 


i 




*a/< T;i'hle of random numbers A; B B L 


11- 


7 


4 


2 


1 




65. Fixed-random-mi3:ed models „ A B ,C_p_E 


} 


4 


6 • 6 


i 1 
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A 

B 

C 

D 

E 



Thoroughly understand 
Fait understanding 
Some understanding 
Have only heard of it 
Have never heard of it 



66. Cause and effect 

57": 2Er3T 

5TT. Hating errors 



Understanding Level 
A B C D E 



cjL ^ 

strong vs. weaR ass umptions 
0 , ue i imitation 



ABODE 
X'B C D E 



A B C P E 
XTIC D E 



'1. 



Confounding design 
.v:i ■ ■ .J 



i 

' 


9 


7 , 1 


\ 


3 


8 

rr* 


6 j 

7 


7 


_1 


1 


5 

“T* 

2 


8 

7 


4 

5 
7 


1 



For the following items, encircle the code letter which best 

Lowledge It tbe^items and indicate whether or not you can do the compute . 
by encircling ?’yes" or "no." 

Understanding Level Computation 



rt 

■ m 



ERIC 
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4 



A - Thoroughly understand 
B - Fair understanding 
C - Some understanding 
D - Have only heard of it 
E - Have never heard or it 




i 

1 
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SUMMARY 

of 

INSTITUTE EVALUATION FORM 



Statemetits 



SESSION ON PERT 

Strongly 
Agree Agree 



Strongly 

Undecided Disagree Disagree 



1. 


The purposes of the Institute 
were clear to me. 


10 


U 




2. 


The objectives of this 
Institute were not realistic. 








3. 


Specific purposes made it easy 
to work efficiently. 


4 


14 




4. 


The participants accepted 
the purpose of the Institute. 


8 


9 


1 


5. 


The objectives of this program 
were not the same as my objectives. 




1 


6. 


I didn't learn anything new. 








7. 


The material presented 
was valuable to me. 


16 


2 




8. 


I could have learned as much 
by reading a book. 








9. 


Possible solutions to my 
problems were considered. 




15 


3 


10. 


The information presented 
was too elementary. 








11. 


The instructors really 
knew their subject. 


16 


2 




12. 


I was stimulated to think 
objectively about the topics 
presented. 


9 


9 




13. 


New acquaintances were made 
which will help in future 
research. 


7 


9 


2 



7 11 



8 9 

3 15 



7 11 



11 7 
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INSTITUTE EVALUATION FORLI (Cont’d) 



Statements 



Strongly 
Agree Agree 



14. 



We worked together as a 
group . 



15. 

16. 
17. 



We did not relate theory 
to practice. 



The sessions followed a 
logical order. 



The schedule was too fixed. 



18. There ii?as very little time 
for informal conversation. 



19. 

20 . 



I did not have the opportunity 
to express my ideas . 



I really felt a part of this 
group . 



21 . 

22 . 



My time was well spent. 



The Institute met my expecta- 
tions. 



23. I received no guide for further 
action. 



24. Too much time was devoted to 
trivial matters 



25. The information presented 
was too advanced. 



26 



The content presented was not 
applicable to the work I do. 



27 



Institutes of this nature 
should be offered again in 
the future. 



15 



! 0 , 



Institutes such as this will 
contribute little to educational 
research and development . 



|E 

1 “ 



er|c 



14 



Undecided 

2 



Strongly 
Disagree Disagree 



10 



14 



9 

13 



12 



16 

11 



11 

1 



7 

11 



12 



1 

7 



11 

7 



14 



f- 



1 



